Pipelined Training with Stale Weights in Deep Convolutional Neural Networks

The growth in size and complexity of convolutional neural networks (CNNs) is forcing the partitioning a network across multiple accelerators during training pipelining backpropagation computations over these accelerators. Pipelining results use stale weights. Existing approaches to pipelined avoid or limit weights with techniques that either underutilize increase memory footprint. This paper contributes scheme uses maximize accelerator utilization keep overhead modest. It explores impact on statistical efficiency performance using 4 CNNs (LeNet-5, AlexNet, VGG, ResNet) shows when introduced early layers, converges models comparable inference accuracies those resulting from nonpipelined (a drop accuracy 0.4%, 4%, 0.83%, 1.45% for networks, respectively). However, deeper network, significantly (up 12% VGG 8.5% ResNet-20). also hybrid combines address this drop. potential improvement proposed demonstrated proof-of-concept implementation PyTorch 2 GPUs ResNet-56/110/224/362, achieving speedups up 1.8X 1-GPU baseline.